User Testing 


User testing and user research can help us to make informed decisions for both iteration and 
product direction. User experience (UX) consists primarily of user testing and user research, 
which are most effective when used in-tandem with QA and analytics tracking. 

UX is by no means meant to exist as the sole form of evaluation, and should work alongside 
and complement data collection and analytics. UX, by nature is primarily qualitative, which is in 
fact one of its main strengths. Where analytics can help us understand how something is 
performing -- and even show us the exact pitfalls in a specific Ul — it can be much harder to 
interpret a clear next step without thinking like one of our users or hearing from a user directly. 

By balancing our evaluations and experimentations between analytics and UX methodologies 
we can get a better understanding of both our overall user-base and a subset of our individual 
users. The goal of this balance is to be able to make informed decisions, keep track of our 
experiments, evaluate and define next steps all while keeping our users in the forefront. 



Proposed Usability Protocol 

Ordered by where in the design process testing should occur 


1. Discount Usability Testing 

When: Pre-user testing, when shipping out a new feature 
What: Heuristic evaluations or task analysis performed internally 
Time: Half of a day 

2. Accessibility Testing (Automated) 

When: 

• While evaluating existing pages 

• Before shipping out a new feature, re-design or page 
What: Online health check or keyboard test 

Time: Half of a day to test and propose changes 

3. Guerrilla Testing 

When: 

• Post-internal analysis, when shipping out a new feature 

• As exploratory research with mocks, storyboards or prototypes 

• Can be used to complement analytics or A/B testing 
What: 5-10 minute interviews or surveys with 25 external users 
Time: Typically 1-3 days to set-up test, run test and evaluate test 

4. User Interviews (Formal Usability Testing) 

Can be performed throughout the lifecycle of a project 
When: 

• Can be used to complement analytics or A/B testing 

• Before starting work on a new re-design or feature 

• To help us evaluate what might need to be changed in an interface 

• The ultimate tiebreaker, why be stuck in a debate when we can talk to users? 
What: 1 5 minute to hour long interviews with 5-1 0 users 

Time: Typically 2-3 days to set-up test, run test and evaluate test + recruiting time 

5. Extended Studies 

When: Before shipping a major change or after for evaluation purposes 
What: Diary studies or other extended field studies 
Time: 1 week to run, half of a week for evaluation 


Discount Usability Testing 

Read More: http://www.nnqroup.com/articles/discount-usabilitv-20-vears/ 

http://www.nnqroup.com/articles/summarv-of-usabilitv-inspection-methods/ 

Tools: Heuristic Evaluations, Cognitive Walkthroughs, Participatory Walkthroughs, Storyboarding 

“Discount Usability Testing,” although ‘free’ can be very powerful and help us to quickly and 
inexpensively ensure that a base level of usability is achieved throughout the product. Discount 
Usability Testing requires an understanding of both UX principles and of our users. 

Discount Usability Testing can take a number of forms, ranging from Internal testing and 
evaluation performed internally to Guerrilla testing. For the purpose of this doc and our protocol 
however, we will be focusing on internal testing only, specifically usability inspection methods 
such as Heuristic Evaluations and Cognitive Walkthroughs. 

These methods rely on having evaluators inspect a user interface with the goal of finding 
usability problems and assessing the severity of these problems. Evaluators can use a shared 
list of heuristics, mental model research or a task to analyze the interface. 

Accessibility Testing 

Read More: http://www.nnqroup.com/articles/kevboard-accessibilitv/ 

http://www.nnqroup.com/articles/bevond-accessibilitv-treatino-users-with-disabilities-as-people/ 

http://www.w3.orq/WAI/ER/tools/ 

Tools: Keyboard testing, WAVE (web accessibility tool) 

In order to best serve all of our users and to ensure equal access for all, it is important that we 
test the accessibility of our websites for users with disabilities. There are a number of online 
tools that we can use quickly uncover where our work falls below accessibility web standards. 
Additionally keyboard tests can help to illuminate the level of difficulty that a user using a screen 
reader might have performing tasks on a specific webpage. 

Guerrilla Testing 

Read More: http://www.uxbooth.com/articles/the-art-of-querilla-usabilitv-testinq/ 

Tools: Surveys (Qualtrics is a powerful survey software that can even be used for A/B testing and heat mapping ), 
paper prototypes, inVision (quick walkthrough prototypes) 

The main principles behind Guerrilla Testing is that it is quick and inexpensive, but also powerful 
because it generates a good amount of qualitative and potentially quantitative data from current 
or potential users. 


Guerrilla Tests can focus on branding, usability or user research and can occur in person or 
online. Utilizing surveys to perform guerrilla testing can make these tests easy to disseminate 
and analyze. 

Users can be compensated with small prizes, raffles, snacks or small cash amounts. 

User Interview (Formal User Testing) 

Read More: http://www.nnqroup.com/articles/thinkinq-aloud-the-1-usabilitv-tool/ 
http://www.nnqroup.com/articles/remote-usabilitv-tests/ 

Tools: Silverback (video and screen capture for iOS), Morae (video and screen capture for PC), Ovo Studios (for 
multiple webcams and screen capture, PC only), UserTestinq.com (for remote testing with new users) 

Formal Usability Testing or User Interviews are used to gain user insight on a flow or an entire 
product. User interviews can either be guided or unguided and can take place in person or 
remotely through a user testing service. 

Guided User Interviews will generally focus on a task or a specific flow of tasks. While 
navigating through the interface and addressing each task, users are generally asked to talk 
aloud. Additionally, testers can decide to stop users between tasks to further evaluate the 
interface. Unguided testing generally also requires users to talk aloud, however the session can 
focus on user exploration rather than being task driven. Generally users fill out a demographic 
survey before beginning a test and an exit survey to assess the user’s overall reaction and 
emotional state after the test or interview. 

All Formal User Testing (both remote and in person) should be recorded with screen capturing 
software, therefore allowing deep analysis. These recordings can also be used to invite the 
larger team to see how users are reacting to the product or a specific interface. 

Generally users are compensated for their time, ranging from $10 for short online tests to $100 
for long in person tests. 


Extended Studies 

Read More: http://www.usabilitvprofessionals.orq/uxmaqazine/dear-diarv-usinq-diaries-to-studv-user-experience/ 
http://punchcut.com/perspectives/uncoverinq-context-mobile-diarv-studies/ 

Extended studies, such as Diary Studies or User Shadowing are used to track a user interacting 
with the product for an extended period of time (generally one week) in a user’s normal 
environment. ForArchive.org this might mean 

For a Diary Study, users are recruited and asked to use the product everyday for a minimum 
amount of time and answer a short survey about using the product either directly after each use 


or simply at the end of their day. The benefits of a Diary Study are the ability to see how the 
product stands up in a variety of different scenarios and locations, especially those where users 
may be distracted. Additionally, this Diary Studies can be used to focus in on a user’s first week 
experiences from onboarding forward. 

Participants are generally compensated for their time and effort, ranging from $25-$100. 



